INTEGRAL OPERATORS ON THE OSHIMA COMPACTIFICATION OF A 
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Abstract. Consider a Ricmannian symmetric space X = G/K of non-compact type, where G 
denotes a connected, real, semi-simple Lie group with finite center, and K a maximal compact 
subgroup of G. Let X be its Oshima compactification, and (n, C(X)) the regular representation 
of G on X. In this paper, a regularized trace for the convolution operators n(f) is defined, 
yielding a distribution on G which can be interpreted as global character of it. In case that / has 
compact support in a certain set of transversal elements, this distribution is a locally integrable 
function, and given by a fixed point formula analogous to the formula for the global character 
of an induced representation of G. 
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1. Introduction 

Let G be a connected, real, semi-simple Lie group with finite center, K a maximal compact 
subgroup, and G/K the corresponding Riemannian symmetric space which is assumed to be of 
non-compact type. In this paper, a distribution character for the regular representation of G on the 
Oshima compactification of G/K is introduced, and a corresponding character formula is proved. 
The paper is a continuation of [8], to which we shall refer in the following as Part I. 

In his early work on infinite dimensional representations of semi-simple Lie groups, Harish- 
Chandra [6] realized that the correct generalization of the character of a finite-dimensional repre- 
sentation was a distribution on the group given by the trace of a convolution operator on represen- 
tation space. This distribution character is given by a locally integrable function which is analytic 
on the set of regular elements, and satisfies character formulas analogous to the finite dimensional 
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case. Later, Atiyah and Bott [3] gave a similar description of the character of a parabolically 
induced representation in their work on Lefschetz fixed point formulae for elliptic complexes. More 
precisely, let H be a closed co-compact subgroup of G, and g a representation of H on a finite 
dimensional vector space V. If T(g) = (i*£>)(g) is the representation of G induced by g in the space 
of sections over G/H with values in the homogeneous vector bundle G x# V, then its distribution 
character is given by the distribution 

9 T : C C °°(G) 3 f —> Tr T(f), T(f) = [ f(g)T(g)d G (g), 

JG 

where d G denotes a Haar measure on G. The point to be noted is that T(f) is a smooth operator, 
and since G/H is compact, does have a well-defined trace. On the other hand, assume that jeG 
acts on G/H only with simple fixed points. In this case, a transversal trace Tr^ T(g) of T{g) can 
be defined within the framework of pseudodiffcrcntial operators, which is given by a sum over fixed 
points of g. Atiyah and Bott then showed that, on an open set Gt C G, 

©t(/) = / /(<?) Tr b T(g)d G (g), / g Cf(G r ). 

J Gt 

This means that, on Gt, the character Qt of the induced representation T is represented by the 
locally integrable function Tr b T(g), and its computation reduced to the evaluation of a sum over 
fixed points. When G is a p-adic reductive group defined over a non-Archimdean local field of 
characteristic zero, a similar analysis of the character of a parabolically induced representation 
was carried out in [S]. 

In this paper, we consider the regular representation n of G on the Oshima compactification X 
of a Riemannian symmetric space X = G/K of non-compact type given by 

n(gMx) = pig- 1 ■ x), <p g C°°(X). 

Since the G-action on X is not transitive, the corresponding convolution operators 7r(/), / g 
C£°(G), are not smooth, and therefore do not have a well-defined trace. Nevertheless, it was shown 
in Part I that they can be characterized as totally characteristic pseudodifferential operators of 
order — oo. Using this fact, we are able to define a regularized trace Tr re9 7r(/) for the operators 
7r(/), and in this way obtain a map 

e 7r :C^(G)9/^Tr res (/)eC, 

which is shown to be a distribution on G. This distribution is defined to be the character of the 
representation it. We then show that, on a certain open set G(X) of transversal elements, 

Tr reg tt(/) = f f(g) Tr b v{g)d G {g), f g C C °°(G(X)), 
Jg(x) 

where, with the notation 3> g (x) — g ■ x, 

Tr ^ (5) % 6 £ 9) idetaJv>(s))r 

the sum being over the (simple) fixed points of g g G(X) on X. Thus, on the open set G(X), 0„. is 
represented by the locally integrable function Tr ir(g), which is given by a formula similar to the 
character of a parabolically induced representation. It is likely that similar distribution characters 
could be introduced for G-manifolds with a dense union of open orbits, or for spherical varieties, 
and that corresponding character formulae could be proved. 

This paper is structured as follows. In Section^ the main results of Part I that will be needed 
in the sequel are recalled. The regularized trace for the convolution operators 7r(/) is defined 
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in Section [3l while the transversal trace of a pscudodiffcrcntial operator is introduced in Section 
HI followed by a discussion of the global character of an induced representation. After studying 
G-actions on homogeneous spaces in Section [5j we prove that the distribution O^ is regular on 
the set of transversal elements G(X), and given by the locally integrable function Tr^^). This 
is done in Section [6l In the last section, the Oshima compactification of X = SL(3,K)/SO(3) is 
described in detail. 

2. Preliminaries 

In this section we shall briefly recall the main results of Part I relevant to our purposes. Let 
G be a connected, real, semi-simple Lie group with finite centre and Lie algebra g, and denote by 
(X, Y) = Tr (a,dX o ad F) the Cartan- Killing form on g. Let 9 be a Cartan involution on g, and 

g = «©p 

the corresponding Cartan decomposition. Put {X. Y)g := — (X, 9Y). Consider further a maximal 
Abelian subspace a of p. Then ad (a) constitutes a commuting family of self-adjoint operators on 
g relative to (,)g, and one defines for each a £ a* the simultaneous eigenspaces g Q = {X £ g : 
[H,X] = a(H)X for all H £ a}. Let £ = {a £ a* : a ^ 0,g Q ^ {0}} be the set of roots of 
(g, a), S + — {a £ X : a > 0} a set of positive roots, and A = {ai, . . . , a;} the set of simple roots. 
Define n + = © aeS + g", n~ = 0(n + ), and write K, A, N + and N~ for the analytic subgroups 
of G corresponding to t, a, n + , and n~, respectively. Let M and M* be the centralizer and 
the normalizer of a in K, respectively. Consider then the Oshima compactification X of the 
Riemannian symmetric space X = G/K which is assumed to be of non-compact type. It is a 
simply connected, compact, real-analytic manifold without boundary carrying a real-analytic G- 
action. The corresponding orbital decomposition of X is of the form 

(1) X~ |J 2# e {G/P e (K)) (disjoint union), 

ecA 

the union being over subsets of A, where #0 is the number of elements of O, and 2# e (G/Pe(K)) 
denotes the disjoint union of 2# e copies of the homogeneous space Gj Pq(K), where Pq(K) is a 
certain closed subgroup of G associated to 0. In particular, for = A, one has Pa(K) = K, 
while for = 0, P$(K) = P. In what follows, denote by Xe a component in X isomorphic to 
G/P®(K). The orbital decomposition is of normal crossing type, meaning that for every point in 
X there exists a local coordinate system (m, . . . ,nk,ti, . . . ,t{) in a neighbourhood of that point 
such that two points with coordinates (n\, . . . , rife, fi, . . . , t{) and (n[, . . . , n' k , ti, . . . , t' { ) belong to 
the same G-orbit if, and only if, sgni,- = sgn^- for all j = 1, . . . , /. For a detailed description of X, 
the reader is referred to Part I. On X, there is a natural representation of G given by 

n(g)<p(x) = (fiig^ 1 ■ x), (p £ C(X), 

where C(X) denotes the Banach space of continuous, complex-valued functions on X. Let da be a 
Haar measure on G, and 5(G) the space of rapidly decreasing functions on G introduced in Part 
I, Definition 1. Let further ft be the density bundle on X, and consider for every / £ S(G) the 
continuous linear operator 

tt(/) : C°°(X) — ■» C°°(X) C X>'(X), 
with Schwartz kernel given by the distributional section JCf £ Z>'(X X X, 1 IE1 Q). Let 

l. J wEW 
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be the finite atlas of X constructed in Part I, where W = M*/M denotes the Weyl group of (g, o), 
and m w G M*, a representative of w G W. The coordinates on each of the charts of this atlas are 
then precisely of the form (n, t) = (m, . . . , Uk,ti, ■ • ■ ,ti) described above. For each point x G X, 
choose open neighborhoods Wx C W~ of x contained in a chart U m ix)- Since X is compact, we can 

find a finite subcover of the cover I W$ \ _, and in this way obtain a finite atlas •! W~, tp^ 1 \ 

I J sex I 1 ) yei 

of X, where for simplicity we wrote tp 1 = ip mm ^)- Further, let {a-y}^ Gl be a partition of unity 

subordinate to the above atlas, and let {a 7 } 7g7 be another set of functions satisfying a 7 € C^'{W! 1 ) 

and Qf 7 |^y =1- Consider now the localization of ir(f) with respect to the atlas above given by 



A}u= [7r(/),^(« o p- 1 )] o ^, ue C~(wg, Wy = <p- 1 (W 1 )c R k+l . 
Writing tp^ = ip^ 1 o g^ 1 o ip 1 and x = {x\, . . . , a;fc+j) = (n, i) G W 7 we obtain 

A]u(x) = y /(sJKaoy^ftjKf 1 • <p y (x)) d G (g) = J f(g)c J (x,g)(uo (p3)( x )d G (g), 

where we put c 7 (x, <?) = a 7 (t;~ 1 • ip 7 (x)). Next, define the smooth functions 

(2) a}(x,0 = J G e^^<c,(x,g)f(g)d G (g), 

and let T x be the diagonal (I x /)-matrix with entries x k +i, • ■ • , Xk+i ■ Introduce the auxiliary symbol 

(3) ~a}(x,0=a}(x,(l k ®T~ 1 )0= [ e^^c 7 (x, g)f(g)d G (g), 

J G 

where we put 

®-y{g, x ) = i( 1 k®T^ 1 )(ip? f (x)-x)} = (x 1 (g-x)-x 1 (x), x k (g-x)-x k {x),x 1 (g, ...,x t (g, x)-l), 

the Xj{gi%) being analytic functions, see Part I, equation (28). One of the main results of Part I 
is the following 

Theorem 1. Let f G <5(G). The restrictions of the operators 7r(/) to the manifolds with cor- 
ners Xa are totally characteristic pseudodifferential operators of class L^ 00 . More precisely, the 
operators 7r(/) are locally of the /orm0 

(4) A}u(x) = J e ix <a}(x,Z)u(Z)dt, u G C^°(W 7 ), 

where aj(x,€) = 5j(x,£i, . . . ,£k,%k+i£k+i, • • • ,€k+l%k+l), and aj(x,£) G S 1 ^ 00 (W 7 x is a 

lacunary symbol given iii/ Q. In particular, the kernel of the operator AJ is determined by its 
restrictions to W* x W* , where W* — {x G W 7 : x k +i ■ • ■ Xk+i ^ 0}, and given by the oscillatory 
integral 



(5) K A] (x,y)= J e^-y^a}(x,0dt. 

Proof. See Part I, Theorem 2. □ 



^Here and in what follows we shall adhere to the convention that, if not specified otherwise, integration is to be 
performed over whole Euclidean space R", with n appropriate. In addition, we shall use the notation <ft; = (2ir)~ n d£. 
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3. Regularized traces 

We shall now define a regularized trace for the convolution operators 7r(/) introduced in the 
previous section. To begin with, note that, as a consequence of Theorem[TJ we can write the kernel 
of 7r(/) locally in the form 

K A] (x >y ) = f e^-y>ta}(x,0<%= f e^-yy^^^Oldetil^T- 1 )'^ 

(6) 1 - 

= , :AVx,Xl — J/1, . • . , 1 —,•••)) Zfc+l • • • Xk+l 

\x k +i ■ ■ ■ X k+ l\ 1 X k+ i 

where AJ(x,y) denotes the inverse Fourier transform of the lacunary symbol aj(x,(;) given by 

(7) A}(x,y) = J e*»*(P f (x,t)dZ. 

Since for x 6 W 7 , the amplitude aj(x, £) is rapidly falling in £, it follows that y) 6 S(Ry ), the 

Fourier transform being an isomorphism on the Schwartz space. Therefore, K A t ( x i y) is rapidly 
decreasing as \xj\ — >• for xj =/= yj and fc + 1 < j < k + I. Furthermore, by the lacunarity of a^, 
K a~i (x, y) is also rapidly decaying as \yj \ — > 0, Xj yj and k + l<j<k + l. 

Consider now the partition of unity {a 7 } subordinate to the atlas {(W 7 , ip^ 1 )}. By equation 
, the restriction of the kernel of A J to the diagonal is given by 

K A i(x,x) = tAI(x,Q), x k +i ■ --Xk+i ^ 0. 

f \x k+x ■ ■ ■ X k +l\ 1 

These restrictions yield a family of smooth functions kj(x) — K A -r((p~ (x),(p~ (x)) which define 
a density kf on 

2*\G/K) c X. 

Nevertheless, the functions kl(x) are not locally integrable on the entire compactification X, so 

that we cannot define a trace of ir(f) by integrating the density kf over the diagonal A^ x ^ ~ X. 
Instead, we have the following 

Proposition 1. Let f 6 S(G), seC, and define for Res > 

Tr s 7r(/)=Y] / (a 1 otp 1 )(x)\x k+ i---x k+ i\ s A 1 Ax,Q)dx 
~ Jw., 

= (^\xk+i ■ • •x fc+ ;| s ,^(a 7 o </j 7 )A}(-,0)^ . 

Then Tr s n(f) can be continued analytically to a meromorphic function in s with at most poles at 
— 1, —3, .... Furthermore, for s 6 C — { — 1, —3, . . .}, 

(8) e£:C c °°(G)9/^Tr s 7r(/)eC 
defines a distribution density on G. 

Proof. The fact that Tr s 7r(/) can be continued meromorphically is a consequence of the analytic 
continuation of \x k +\ ■ ■ ■ x k+ i\ s as a distribution in R fe+i , proved by Bernshtein-Gel'fand in 0], 
Lemma 2. One even has that 

(\x k+1 \ s i---\x k+l \ s ',u), ueC™(R k+l ), 

can be continued meromorphically in the variables s\, . . . , s; to C' with poles s,: = —1, —3, .... 
To see that (jHJ is a distribution density, note that : C£°(G) — > C is certainly linear. Since 
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\xk+i ■ ■ ■ Xk+i\ s is a distribution, for any open, relatively compact subset oj C R k+l there exist 
C w > and B u G N such that 

(9) \(\x k+1 ---x k+l \ s ,u)\ <C U 2 Bup|^«|, u€Cf(w). 

\P\<B U 

Let now C G be an arbitrary open, relatively compact subset, and / £ C£°(0). With equation 
(J7]) one has 

(!0) Tr s ?r(/) = ^|a; fc+ i • • ■ x k+ i\ s ,^(a^ o ip 7 ) J a}(-,g)d£j . 

By equation (38) of Part I, one computes for arbitrary N € N that 

27V 

e^^-e = — - I E b»(x,g)dL(X«)[e^->^< 

{ + ISM r=0 | a | =r , 

where the coefhcients 6^ (a;,g) are smooth, and at most of exponential growth in g. With © and 
Proposition 1 of Part I we therefore obtain for aj(x,£) the expression 

1 f 2N r i 

*}&0= ri + i f |2w / e^^^EE^W^) ^9)<h(x,g)f{g) d G (g). 

Inserting this in (fT0|) . and taking sufhciently large, we obtain with (|9]) that 

|Tr s7 r(/)| <C ^ sup|dL(J^)/| 

|/3|<B 

for suitable Co > and -Be> £ N. Since the universal enveloping algebra il(rjc) can be identified 
with the algebra of invariant differential operators on G, the assertion now follows with 9 , page 
480. " ' □ 

Remark 1. Using Hironaka's theorem on resolution of singularities, Bernslitein-Gel'fand [I] and 
Atiyah [T] even proved the following general result. Let M be a real analytic manifold and / a 
non-zero, real analytic function on M . Then |/| s , which is locally integrable for Res > 0, extends 
analytically to a distribution on M which is a meromorphic function of s in the whole complex 
plane. The poles are located at the negative rational numbers, and their order does not exceed the 
dimension of M. From this one deduces that if / : M — > C is a non-zero analytic function, then 
there exists a distribution S on M such that fS = 1. This is the Hormander-Lojasiewicz theorem 
on the division of distributions, and implies the existence of temperate fundamental solutions for 
constant-coefficient differential operators. 

Consider next the Laurent expansion of 0^.(/) at s = — 1. For this, let u € C£°(R fc+ ') be a test 
function, and consider the expansion 

oo 

(\x k +i ■ ■ -x k+ i\ s ,u} = 53 s j( u )( s + 

where S k € V(M. k+l ). Since \x k +i ■ ■ ■ x k +i\ s+1 has no pole at s = — 1, we necessarily must have 

\x k +\ ■ ■ -Xk+i\ ■ Sj = for j < 0, \x k+ i ■ ■ -x k+ i\ ■ S = 1 

as distributions. Therefore Sq € T)'(M. k+l ) represents a distributional inverse of \x k +i ■ ■ ■ x k +i\. By 
repeating the reasoning of the proof of Proposition [1] we arrive at the following 
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Proposition 2. For f € <S(G), let the regularized trace of the operator tt(/) be defined by 



\ 7 / 
Then 8^ : C£°(G) 3 f i-t Tr reff 7r(/) S C constitutes a distribution density on G, which is called 
the character of the representation it. 



Remark 2. An alternative definition of Tr reff 7r(/) could be given within the calculus of b- 
pseudodifferential operators developed by Melrose. For a detailed description, the reader is referred 
to [7], Section 6. 

In what follows, we shall identify distributions with distribution densities on G via the Haar 
measure da- Our next aim is to understand the distributions <d^ and 0^ in terms of the G-action on 
X. We shall actually show that on a certain open set of transversal elements, they are represented 
by locally integrable functions given in terms of fixed points. Similar expressions where derived by 
Atiyah and Bott for the global character of an induced representation of G. Their work is based 
on the concept of transversal trace of a pseudodifferential operator, and will be explained in the 
next section. 



In [2], Atiyah and Bott extended the classical Lefschetz fixed point theorem to geometric en- 
domorphisms on elliptic complexes. Their work relies on the concept of transversal trace of a 
smooth operator, and its extension by continuity to pseudodifferential operators. The Lefschetz 
theorem then follows by showing that the Lefschetz number of a geometric endomorphism is given 
by an alternating sum of transversal traces, and extending an analogous alternating sum formula 
for smooth endomorphisms. To explain the notion of transversal trace of a pseudodifferential 
operator, let us introduce the following 

Definition 1. Let M be a smooth manifold. A fixed point Xo of a smooth map f : M M is 
said to be simple if det (1 — df Xo ) ^ 0, where df Xa denotes the differential of f at xq. The map f 
is called transversal if it has only simple fixed points. 

Note that the non- vanishing condition on the determinant is equivalent to the requirement that 
the graph of / intersects the diagonal transversally at (xq, xq) <E M x M, and hence the terminology. 
In particular, a simple fixed point is an isolated fixed point. Let now U be an open subset of M™, 

V open in U, and consider a smooth map a : V — > U with a simple fixed point at xq- We choose 

V so small, that x M- x — a(x) defines a diffeomorphism of V onto its image. Let A : V —> U x U 
be the map A(x) = (a(x),x), and assume that A 6 L~°°([/) is a smooth operator with symbol 
a(x,£). The kernel K a of A is a smooth function on U X U, and its restriction A*Ka to the graph 
of a defines a distribution on V according to 



where we made the substitution y = x — a(x), and the change in order of integration is permissible 
because a(x,£,) 6 S~°°([/). Now, for a(x,£) £ S (U), we observe that by differentiating 




□ 



4. Transversal trace and characters of induced representations 



(11) 
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with respect to £, and integrating by parts with respect to y, we obtain the estimate 

i 3? J e-*H«(*m,o |det (tt^U)))! *' - lQ " m 

for arbitrary multi-indices 7 and /3 and some constant C > 0. Thus, as an oscillatory integral, the 
last expression in (TTT|) defines a distribution on V for any a(x,£) £ S (J7). The distribution A*iGi 
is called the transversal trace of A £ l}(U). If, in particular, a(x,£) — a(x) is a polynomial of 
degree zero in £, one computes that 

U2j A |det(l-da(a; ))|- 

This diccussion can be globalized. Let X be a smooth manifold, E a vector bundle over X, 
a : X —y X a C°°-map with only simple fixed points, and 

A : T c {a*E) — > r(F) 

a pseudodiffercntial operator of order Z between smooth sections. Denote the density bundle 
on X by fi, put F = a*E, and define F' = F* ® f2. The kernel -FCi is then a distributional 
section of £ IE F'. In other words, K A £ V (E ® F') = V(X xX,£l F'). Similarly, one has 
K a ,A £ 2?'(X xX,FI F'), where aM denotes the composition 

a* A : r c (F) A T(F) ^ r ( F ). 

If ^4 g L _00 (F, F), Ka is a smooth section on X x X, and Ka{x,d) £ E x ® F' y . In this case, 
K a -A{x,y) — KA(a(x),y), so that one deduces K a *A(x, x) £ Fq,^) ® F^ = F x <g) (F* ® n) a ~ 
C(F X ,F X ) <E) 0, x - As a consequence, Tr K a *A(x, x) becomes a section of O, where Tr denotes the 
bundle homomorphism 

(13) Tr : F <g) F' — >• Q. 

Hence, if X is compact, one can define the trace of a* A as 



Tva*A = / Tr K *a(x, x) 
7x 



This trace can be extended to arbitrary A £ L (X). Indeed, let A be the diagonal in X x X, and 
denote the canonical isomorphism A ~ X also by A. The foregoing local considerations imply that 
the map 9 : C{£ '(F), Y(E)) -> T(F <g> F') given byin- A*FT Q « A = J^-aOe, x) has an extension 

9 : L'(F,F) — ► V'{F®F') 

which is continuous with respect to the strong operator topology on bounded sets of L l (F, E) , see 
[2J, Proposition 5.3. Since the bundle homomorphism (|13[) induces continuous linear maps 

Tr : T(F (g) F') — ► r(fi), Tr : P'(F <g> F') — ► P'(fi), 

where D'(fi) = £>'(X,0) = T c (0* (g> 0)' = T c (l)' = C£°(X)' is the space of distribution densities 
on X, we see that Tr 0(A) can be defined for any A £ L l (F, E) in a unique way. Consequently, for 
compact X, the map L _00 (F, E) — > C, A — > Tra*A has a unique continuous extension 

Tr Q : L Z (F,F) — > C, A 1— > Tr Q A = (TrG(A),l) , 

called the transversal trace of A. In the case that A is induced by a bundle homomorphism 93, it 
follows from flT2j) that 

< 14 > ^ A = E "-< A >' -< A > - w^wv 

the sum being over the fixed points of a on X, see [2J, Corollary 5.4. 
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In the context of representation theory, this trace was employed by Atiyah and Bott in [3J 
to compute the global character of an induced representation. Thus, let G be a Lie group, H 
a closed subgroup of G, and g a representation of H on a finite dimensional vector space V. 
The representation of G induced by g is a geometric endomorphism in the space of sections over 
G/H with values in the homogeneous vector bundle G x^ V, and shall be denoted by T(g) = 
(t*g)(g)- Assume that G/H is compact, and let d,Q be a Haar measure on G. Consider a compactly 
supported smooth function / G CJ?°(G), and the corresponding convolution operator T{f) = 
f G f(g)T(g)d,G(g). It is a smooth operator, and, since G/H is compact, has a well defined trace. 
Consequently, the map 

e T :C^(G)9/^TrT(/)eC 

defines a distribution on G called the distribution character of the induced representation T. On 
the other hand, assume that g G G is such that l g -i : G/H — > G/H, xH n- g~ 1 xH , has only simple 
fixed points. In this case, a transversal trace Tr b T(g) of T(g) can be defined according to 

Tr b T( 5 ) = Tr Vl (r(^)), 

where ip g : I* „!(G x h V) ^ G x h V is the endomorphism associated to T{g) such that 

T ig) = *'s oi j-i. 

and T(ipg) : (G x H V)) -> T(G x H V). Tr b T(g) is given by a sum over fixed points of g, and 

one can show that, on an open set Gt C G, 

(15) T (/)=/ /( 5 )Tr b T( 5 )d G ( ff ), /GC C °°(G T ). 

Thus, the distribution character of a parabolically induced representation of a Lie group G is 
represented on Gt by the transversal trace of the corresponding geometric endomorphism. If G 
is compact, the Lefschetz theorem reduces to the Hermann- Weyl formula by the theory of Borel 
and Weil. It can be interpreted as expressing the character of a finite dimensional representation 
as an alternating sum of characters of infinite dimensional representations. In what follows, we 
shall prove similar formulae for the distributions and defined in the previous section, after 
reviewing some largely known facts about group actions on homogeneous spaces. 

5. Fixed point actions on homogeneous spaces 

Let G be a Lie group with Lie algebra g, H C G a closed subgroup with Lie algebra f), and 
7r : G — > G/H the canonical projection. For an element g G G, consider the natural left action 
l g : G/H — > G/H given by l g (xH) — gxH. Let Ad G denote the adjoint action of G on q. We 
begin with two well-known lemmata, see e.g. [3J, page 463. 

Lemma 1. l g -i : G/H — > G/H has a fixed point if and only if g G [J xGG xHx -1 . Moreover, to 
every fixed point xH one can associate a unique conjugacy class h(g,xH) in H. 

Proof. Clearly, 

Ig-i(xH) = xH g~ 1 xH = xH <^=> (g~ 1 x)~ 1 x G H <^=> x^ 1 gx = h(g,x), 

where h(g,x) G H. So l g -i has a fixed point xH if, and only if, g G [J X&G xHx^ 1 . Now, if y G G 
is such that xH = yH, then y = xh for some h G H. This gives us that h(g,y) = y~ 1 gy = 
(xh)~ 1 g(xh) — h~ 1 (x~ 1 gx)h = h~ 1 h(g,x)h. Thus, as x varies over representatives of the coset 
xH, h{g,x) varies over a conjugacy class h{g,xH) in H. □ 
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Lemma 2. Let xH be a fixed point of l g -i and let h £ h(g,xH). Then 

det(l-dZ s -i) xH = det(l-Adg(/i)), 
where Ad^ : H — > Aut(g/(i) is the isotropy action of H on g/f). 

Proof. Let L g and R g be the left and right translations, respectively, of g E G on G. We begin 
with the observation that 

(16) 7r o L g -i = l g -i o 7T, 

where 7r is the natural map from G to G/H. Let e be the identity in G, and T^^iG / H) the 
tangent space to G/ff at the point 7r(e). The derivative e?7r : g — > T 7r ( e )(G/i?) is a surjective linear 
map with kernel f), and therefore induces an isomorphism between g/f) and T^^{G / H), which we 
shall again denote by dn. Notice also that, for h £ H, Ad (h) leaves f) invariant and so induces a 
map Adjy(/i) : g/fj — > g/f). Now, let xH be a fixed point of l g -i, and take h £ h(g,xH). Choose x 
in the coset xH such that g~ x x = xh. For y e G one computes 

(noL g -i oR h -i)(y) = TrCg" 1 ^ 1 ) = .g^ytf = l g -x{yH) = (7 g -i o7r)(y), 

so that 

(17) 7T O L g -1 O = O 7T. 

Observe, additionally, that L g -i o fixes x. We therefore see that L g -i o o L x = L x o Lh ° 
Rh-i, which, together with equations pB| and (|17p . leads us to 

T n r x )(G/H), we obtain the 



□ 

Consider now the case when G is a connected, real, semi-simple Lie group with finite centre, 9 & 
Cartan involution of g, and g = t © p the corresponding Cartan decomposition. Further, let K be 
the maximal compact subgroup of G associated to t, and consider the corresponding Riemannian 
symmetric space X = G/K which is assumed to be of non-compact type. By definition, 9 is an 
involutive automorphism of g such that the bilinear form (•, -)g is strictly positive definite. In 
particular, (•, -)g|pxp is a symmetric, positive-definite, bilinear form, yielding a left-invariant metric 
on G/K. Endowed with this metric, G/K becomes a complete, simply connected, Riemannian 
manifold with non-positive sectional curvature. Such manifolds are called Hadamard manifolds. 
Furthermore, for each g 6 G, L-i : G/K — > G/K is an isometry on G/K with respect to this 
left-invariant metric. Note that Riemannian symmetric spaces of non-compact type are precisely 
the simply connected Riemannian symmetric spaces with sectional curvature n < and with no 
Euclidean de Rham factor. We then have the following 

Lemma 3. Let g € G be such that l g -i : G/K — > G/K is transversal. Then l g -i has a unique 
fixed point in G/K . 



(18) l x O 7T O Lh O R h -1 = l g -l O l x O 7T. 

Differentiating this, and using the identification dl x o dn : g/f) 
commutative diagram 

fl/fl > 9/f) 

dl x od-K dl^odn 



dl 



T Ax) (G/H) — 2— ► T n(x) (G/H) 

thus proving the lemma. 
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Proof. Let M be a Hadamard manifold, and ip an isometry on M that leaves two distinct points 
x,y G M fixed. By general theory, there is a unique minimal geodesic 7 : R — > M joining x and 
y. Let 7(0) = x and 7(1) = y, so that ip o 7(0) = tp(x) = x and tp o 7(1) = y>(y) = y. Since 
isometries take geodesies to geodesies, ^07 is a geodesic in M, joining x and y. By the uniqueness 
of 7 we therefore conclude that ^07 = 7. This means that an isometry on a Hadamard manifold 
with two distinct fixed points also fixes the unique geodesic joining them point by point. Since, by 
assumption, l g -i : G/K — > G/K has only isolated fixed points, the lemma follows. □ 

In what follows, we shall call an element g G G transversal relative to a closed subgroup H if 
lg-i : G/H — > G/H is transversal, and denote the set of all such elements by G(H). 

Proposition 3. Let G be a connected, real, semi-simple Lie group with finite centre, and K a 
maximal compact subgroup of G. Suppose rank(G) = rank(A'). Then any regular element of G is 
transversal relative to K. In other words, G' C G(K), where G' denotes the set of regular elements 
in G. 

Proof. If a regular element g is such that l g -i : G/K — > G/K has no fixed points, it is of course 
transversal. Let, therefore, g G G' be such that l g ~i has a fixed point xqK. By Lemma[TJ g must be 
conjugate to an element k(g, xo) in K . Consider now a maximal family of mutually non-conjugate 
Cartan subgroups J\, . . . , J r in G, and put J- = Jj fl G' for i G {1, . . . , r}. A result of Harish 
Chandra then implies that 

G'=\J \JxJ[x-\ 

i=l x£G 

see [9], Theorem 1.4.1.7. From this we deduce that 

g = xk(g,Xo)x~ 1 = yjy 1 for some x,y G G,j G J[ for some i. 

Hence, k(g, xq) must be regular. Now, let T be a maximal torus of K . It is a Cartan subgroup of K , 
and the assumption that rank(G) = rank(iT) implies that that T is also Cartan in G. Let k(g, XqK) 
be the conjugacy class in K associated to xqK, as in Lemma [1] As K is compact, the maximal 
torus T intersects every conjugacy class in K. Varying xq over the coset XqK, we can therefore 
assume that k(g,xo) G k(g,x K) n T. Thus, we conclude that k(g,xo) G T n G'. Note that, in 
particular, we can choose Ji = T by the maximality of the Jx,...,J r . Now, for a regular element 
h G G belonging to a Cartan subgroup H one necessarily has det (1 — Ad^(/i)) ^ 0, compare 
the proof of Proposition 1.4.2.3 in ,9|. Therefore det (1 — Ad^(fc(g, xq))) / 0, and consequently, 
det (1 — Ad x{k(g, xo))) 7^ 0. The assertion of the proposition now follows from LemmaU □ 

Corollary 1. Let G be a connected, real, semi-simple Lie group with finite centre, K a maximal 
compact subgroup of G, and suppose that rank (G) = rank (K) . Then the set of transversal elements 
G(K) is open and dense in G. 

Proof. Clearly, G(K) is open. Since the set of regular elements G' is dense in G, the corollary 
follows from the previous proposition. □ 

Remark 3. To close this section, let us remark that with G as above, and P a parabolic subgroup 
of G, it is a classical result that G' C G(P), see [5], page 51. 

6. Character formulae 

Let the notation be as before. We are now in a position to describe the distributions Q% and 
07r introduced in Section [3] Thus, let (vr, G(X)) be the regular representation of G on the Oshima 
compactification X of the Riemannian symmetric space X = G/K of non-compact type, and denote 
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by & g (x) — g ■ x the G-action on X. Let further G(X) C G be the set of elements g in G acting 
transversally on X. 

Remark 4. The set G(X) is open. Corollary [T] and Remark [3] imply that G(X) is dense if 
rank (G/K) = 1, and non-empty if rank(G / 'K) = 2, and rank(G) = rank (if). 



(19) Tr s n(f) = f(g) ^ ^ d G ( 5 ), 



Theorem 2. Let / 6 C^°(G) /iaue support in G(X), and s G C, Res > -1. 27i 

-i^" 1 ^)) ■ --Xk+iin 
|det(l-d$ -i(J))| 

where Fix(<7) denotes the set of fixed points of g on X. In particular, 0* : C£°(G) 9 / — > Tr s 7r(/) £ 
C is regular on G(X). 

Proof. By Proposition [TJ 

Tr s 7r(/)=V' / (a 7 o (p 7 )(x)\x k +i ■ ■ ■ x k+ i\ s Al(x,0)dx 
7 Jw-, 

is a meromorphic function in s with possible poles at —1, —3, .... Assume that Res > — 1. Since 
a 7 G GfiW^), and Aj(x,0) = J a}(x,g)d$, where a}(x,£) G S ia oc (W 7 x JR fc+i ) is rapidly decaying 
in £ by Theorem [TJ we can interchange the order of integration to obtain 

Tr s 7r(/)=^/ / (a 1 o(p 7 )(x)\x k+ i---Xk+i\ s aj(x,£)dxd£. 
7 ■/ •'VP, 

Let x G C£°(R fc+ ',R + ) be equal 1 in a neighborhood of 0, and e > 0. Then, by Lebesgue's theorem 
on bounded convergence, 

TtstK/) = limle, 

where we defined 

I e= S ^ J I \ (a^oip 1 )(x)\x k+ i---x k+ i\ s aJ(x,^)x{e^)dxd^. 
7 «/ •'W' 7 

Taking into account ([3]), and interchanging the order of integration once more, one sees that 

I f(9)Y, I I e l ^ x ^c 7 (x,g)(a 1 o^)(x)\x k+1 ---x k+l \ s x(eOdx^d G (g), 
Jg 7 J Jw~, 

everything in sight being absolutely convergent. Let us now set 

h{g) = f{g)Y J [ f e i ^ x >%(x,g)(c^o^)(x)\x k+1 ---x k+l \ s X (eOdx^, 
7 J -iw., 

so that I e — J G I e (g) dc(g)- We would like to pass to the limit under the integral, for which 
we are going to show that lirm-^o I e (g) is an intcgrablc function on G. For this, let us fix an 
arbitrary g G G(X). By definition, g acts only with simple fixed points on X. Since each of them is 
isolated, g can have at most finitely many fixed points on X. Consider therefore a cut-off function 
P g G C°°(X,R + ) which is equal 1 in a small neighborhood of each fixed point of g, and whose 
support decomposes into a union of connected components each of them containing only one fixed 
point of g. By choosing the support of f3 g sufficiently close to the fixed points we can, in addition, 
assume that 

(20) det (d$ g (x) -1)^0 on supp^ s . 
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Since the action of G is real analytic, we obtain a family of functions (3 g (x) depending analytically 
on g € G(X). Multiplying the integrand of I e (g) with j3 g o Lp 1 (x), and 1 — (3 g o (p 1 (x), respectively, 
we obtain the decomposition 

I e (g) = lP(g) + 4 2 \g). 

Let us first examine what happens away from the fixed points. Integrating by parts 2N times with 
respect to £ yields 



7 J Jw 1 



= /(ff)2^ J J w |^( g x )\2n \ [xieO]^^)^^- P a )){fj{x))\x k+1 ■■■x k +i\ s dxd£, 

where = 9^ + • • • + d^ k+l ■ Now, for arbitrary N, 

\A»[ X (eO]\<C N (l + \tf)- N , 

where Cn does not depend on e for < e < 1. Furthermore, there exists a constant Mf > such 
that \ty-y(g, x)\ 2N > Mf on the support of 1 — j3 g o ip^ for all g G supp/ and 7. By Lebesgue's 
theorem, we may therefore pass to the limit under the integral, and obtain 

lim/j 2) (a) = 0. 

Hence, as e — > 0, the main contributions to I e (g) originate from the fixed points of g. To examine 
these contributions, note that condition (|20[) implies that x <— ¥ tp!l(x) — x defines a diffeomorphism 
on each of the connected components of supp(a 7 /3 g ) o <p 7 onto their respective images. Performing 
the change of variables y — x — tp® (x) we get 

x d£ 



-i{±H®T- x )y-(L, , v (Q- 7 ^ g )(y 7 (a;(y)))c 7 (a;fa),g) 



7 J Jw-, 

7 

. nr f, , « M|s , M ( Q!7 ^)(y 7 ( a ;(y)))x((l fc T^y / e) 

= /(ff)l rf y l*H-l&/)-*w(y)| Cr(*(l/),fl) (27r)fe+;£ , +i | det(1 _^ (x(2/))) | * 

xv-y, , , / n i s / / -i > K/3 3 )(y 7 (x( £?/ )))x((i fc r- ( ^ } )y) 

Since in a neighborhood of a fixed point i of g the Jacobian of the singular change of coordinates 
z = (l fe ®T~}.)y converges to the expression |a;fe + i(ft 7 1 (x)) • • • x k +i{n~ 1 (x))\~ 1 as e -> 0, we 
finally obtain with (2-7r)~ fc ~ ( J x(y)dy = x(0) = 1 that 

lim/W( ff ) = lim/( ff ) 



a 7 (x)|xfe+i(^ 1 (^)) ' ■■ x k+i(n 7 1 (x))\ s+1 



= » J^_, V |det (1-^(5))! 



5GFix(g) 7 
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since a 7 = 1 on suppa 7 , and /3 g (x) — 1. The limit function lim 6 _>.o le (<?) is therefore clearly 
integrable on G for Re s > — 1 , so that by passing to the limit under the integral one computes 

Tr s tt(/) = lim/ e = lim ( I £ (g) d G (g) = [ lim (/J 1 ) + I^)(g)d G (g) 



G 



a 1 (x)\x k +i(K 1 1 (x)) ■ ■ -x k +i{n 7 1 (x))| s+1 



= lc m ^£ |det(l -<i* s -i(i))| **>■ 

yielding the desired description of 9^.. □ 

As an immediate consequence of the previous theorem, we see that if / G C£°(G(X)), Tr s 7r(/) 
is not singular at s = — 1. This observation leads to the following 

Corollary 2. Let f G C£°(G) have support in G(X). 77ien 

Tr reg 7r(/) = Tr_i7r(/) = / /(g) V — — — T^-dcO)- 

In particular, the distribution Q n : f — > Tr re9 (/) is regular on G(X). 
Proof. Consider the Laurent expansion of Q^(f) at s = — 1 given by 

(a 7 o 93 7 )a}(-,o)j(s + iy, 

\ 7 / J=-« 7 

where 5^ G £>'(R fe+ '). Since by (|19l) . Tr s 7r(/) has no pole at s = — 1, we necessarily must have 

S^Ko^jijM)) -0 forj<0, 

7 

so that 

Tr_i tt(/) = ^s 0) X)K ° °)y = Tr -9 *(/)• 

The assertion now follows with the previous theorem. □ 

In particular, Corollary [5] implies that Tr res 7r(/) is invariantly defined. Now, interpreting 7r(o/) 
as a geometric endomorphism on the trivial bundle £ = XxC over the Oshima compactification 
X, a transversal trace Tr 1 ' ir(g) of 7r(g) can be defined according to 

Tr b 7r( 5 ) = Trs^^r^g)), 

where </? g : — > E is the associated bundle homomorphism which identifies the fiber E$ _ 1 ^ 
with E^, and satisfies (Tr<^ 9 )ij = 1 at each fixed point x of g. Taking into account (I14p . the 
previous corollary can be reformulated, and we finally deduce the following character formula for 
the distribution character of n. 

Theorem 3. On the set of transversal elements G(X), the distribution 0^ : / — > Tr reg (/) is given 
by 

Tr re3 Tr(f) = f f{g) Tr b v{g)d G {g), f G C C °°(G(X)), 
Jg{x) 

where 

Tr ^) = E |dfit(l-dVi(i))l' 

£SFix(g) 1 V g \ JJ\ 

the sum being over the (simple) fixed points of g G G(X) on X. 
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□ 



7. The case X = SL(3,R)/SO(3) 



We shall finish this paper by describing in detail the Oshima compactification of the Riemannian 
symmetric space X = SL(3, R)/SO(3). Thus, let g = sl(3, R) be the Lie algebra of G. A Cartan 
involution : q — > g is given by X i-> —X t 1 where X 1 denotes the transpose of X, and the 
corresponding Cartan decomposition of q reads q = t © p, where t = {X <E sl(3,R) : X 1 = —X} 7 
and p = {X e s[(3,R) : X 1 = X}. Next, let 

a = {D(ai,a 2 ,a 3 ) : ai,a 2 ,a 3 e R, ai + a 2 + a 3 = 0}, 

where D(ai, a 2 , 03) denotes the diagonal matrix with diagonal elements a\, a 2 and 03. Then a is a 
maximal Abelian subalgebra in p. Define : o — > R by D(ai, a 2 , 03) n> dj, i = 1, 2, 3. The set of 
roots S of (g, a) is given by £ = {±(ej — e^) : 1 < i < j < 3}. We order the roots such that the 
positive roots are S + = {ei — e 2 , e 2 — e 3 , e\ — e^}, and obtain A = {ei — e 2 , e 2 — 63} as the set of 
simple roots. The root space corresponding to the root e\ — e 2 is given by 

fl ei-e 

and similar computations show that 

For a subset 9 C A, let (8) denote those elements of S that are given as linear combinations of 
the roots in 9. Write (9) ± for £± n (9). Put n ± (9) = Eag(o)± A > and n e = T,\eT,+ -(e) + A - 
Let rig = 0(xIq). Consider now the case 9 = {ei — e 2 }. Then n + (ei — e 2 ) = fl Cl ~ e2 , and 
n+ _„„ = g e2 ~ e3 © ei ~ 63 . In other words, 





X 






!(: 





i) 


: x e 





















!(• 





I) 


: z e 























ei-e 3 _ J 


[(: 





t 


1 :2/GrJ 


"1 












ei- 














-1 


[(: 





;) 


: y,z e r| 














Exponentiating, we find that the corresponding analytic subgroups are given by 

N+( ei - e 2 ) = I I 1 ) :*€!*!, iV+ 2 = | ( 1 z]:y,z€ 





In a similar fashion, we obtain that 

n -(ei -e 2 ) = fl C2 - ei = { I x ) :xe 

^«- e2 )=<! |° 0\:y,zG 
\V 

and that the corresponding analytic subgroups read 

'l 0\ 

iV-(ei-e 2 )= < I z 1 :x£l}, AT"_ e2 = { | 1 | : y, z G 
,0 1/ 
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The Cartan-Killing form (-,•) : q x g — > R is given by (X,Y) i-> Ti(XY), and the modified 
Cartan-Killing form by (X,Y) e := -Tr(X0(y)) = -Tr(X(-F*)) = TrpTK*). Next, let a(0) = 
J2\e(e)+ R< 5^' wriere Qa = [0-X,-X] for X e A such that (X,X)g = 1. Also, let a be the 
orthogonal complement of a(0) with respect to (•, -)g. Again, suppose that = {e\ — e 2 }. We find 

Qei — e 2 = 

so that 

















a(ei - e 2 ) = I 




[(: 


— r 


3 


: r e r| 
















This in turn gives us that 

( la 
a, . < () a i : 
[ \0 -2ay 

Exponentiation then shows that the corresponding analytic subgroups are 

( la 0\ ] f /a 

A(ei-e 2 )=< a" 1 :a6l + l,4_ e2 = | a | :a£ B 
l\0 V J lV0 a- 2 , 

Take if = SO(3) as a maximal compact subgroup of SL(3, K), and denote by Mq(K) the centralizer 
of ae in K. Observing that the adjoint action of a matrix group G is just the matrix conjugation, 
we see that 



M ei _ e2 (K) = Z K (a ei - e2 ) 



Notice that M ei - e2 (K) has 2 connected components. Put M = Z K (A), and let P = MAN+ be 
the minimal parabolic subgroup given by the ordering of the roots of (g, a). For G = SL(3,R) one 
computes 

M = 

As varies over the subsets of A, we get all the parabolic subgroups Pq of G containing P, and 
we write P e = M e (K)AN+. By definition, Pq{K) = M @ (K)A e N£ so that, in particular, 




P ei -e 2 (K) = 




/SO(2) 
I 1 




The orbital decomposition of the Oshima compactification X of X = SL(3,R)/SO(3) is therefore 
given by 

X = G/P U 2 • G/P ei - E2 (K) U 2 ■ G/P e2 - e3 (K) U 2 2 • G/K. 
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